1. Field of the Invention
This invention pertains generally to camera depth estimation, and more particularly to depth estimation using picture matching.
2. Description of Related Art
To achieve the most desirable outcome, a camera must be properly focused when capturing an image. Numerous systems have been developed for estimating and/or attaining a proper camera focus. A brief discussion follows of the elements within a typical camera-lens system and their associated characteristics.
Perhaps the principle optical parameters of a photographic lens can be considered to be its maximum aperture and focal length. The focal length determines the angle of view, and the size of the image relative to that of the object (subject) for a given distance to the subject (subject-distance). The maximum aperture (f-number, or f-stop) limits the brightness of the image and the fastest shutter speed usable for a given setting (focal length/effective aperture), with a smaller number indicating that more light is provided to the focal plane which typically can be thought of as the face of the image sensor in a simple digital camera. The focal length of the camera can be single focus or variable, as in response to adjusting a so-called “zoom” control.
One form of simple lens, which is technically a lens having a single element but is typically used more broadly, has a single focal length and is also referred to as a “prime lens”. In focusing a camera using a single focal length lens, the distance between lens and the focal plane is changed resulting in a change in the focus point of the photographic subject onto that plane. Thus, although the single focal length lens has a fixed optical relation and focal length, it is used in the camera to focus on subjects across a range of focus distances. Consequently, one should not confuse the focal distance(s) of a lens with the range of focus obtainable on a camera using that lens, whereby adjusting the position of that lens in relation to the focal plane alters focus distance.
To use a single focal length lens one adjusts the aperture to select the amount of light with respect to desired shutter speed, and then adjusts focus (focuses) according to subject-distance after which the shutter is “released” and the image is captured. It will be appreciated that the trigger for image capture is referred to herein as “shutter release”, although the process of capturing an image in a digital camera is in large measure performed electronically. Often a macro setting is provided with a different focal length selection, on an otherwise single focal length lens, for taking close-up shots. A telephoto lens provides a very narrow angle of view with high magnification for filling the frame with images from distance objects.
Multi-focal length lenses are usually referred to as “zoom” lenses, because image magnification can be “zoomed”, or “unzoomed” as desired to obtain higher, or respectively lower magnification. Zoom lenses allow the user to select the amount of magnification of the subject, or put another way, the degree to which the subject is zoomed to fill the frame. It is important to understand that the zoom function of these lenses, or camera-lens systems, is conceptually separate from both the focus control and the aperture control.
Regardless of whether a single-focal length lens or multi-focal length lens is utilized, it is necessary to properly focus the lens for a given subject-distance. An acceptable range of focus for a given focus setting is referred to as “depth of field” which is a measurement of depth of acceptable sharpness in the object space, or subject space. For example, with a subject distance of fifteen feet, an acceptable range of focus for a high definition camera may be on the order of inches, while optimum focus can require even more precision. It will be appreciated that depth of field increases as the focusing moves from intermediate distances out toward “infinity” (e.g., capturing images of distant mountains, clouds and so forth), which of course at that range has unlimited depth of field.
For a single focal length lens at a given aperture setting, there exists a single optimum focus setting for a given camera to subject distance (referred to as subject-distance). Portions of the subject which are closer or farther than the focus distance of the camera will show up in the captured images subject to some degree of blurring, the extent of which depends on many factors that impact depth of field. However, in a multi-focal lens there is an optimum focus point for each lens magnification (lens focal length) obtainable by the lens. To increase practicality, lens makers have significantly reduced the need to refocus in response to zoom settings, however, the necessity for refocusing depends on the specific camera-lens system in use. In addition, the aperture setting can require changing in response to different levels of zoom magnification.
Originally, camera focus could only be determined and corrected in response to operator recognition and manual focus adjustments. However, due to the critical nature of focus on the results, focusing aids were readily adopted. More recently, imaging devices often provide the ability to automatically focus on the subject, a function which is generically referred to today as “auto focus”. Cameras often also provide a combination of automatic and manual controls, such as the use of aperture priority and shutter priority modes. Focus continues to be a point of intense technical development as each of the many existing auto focus mechanisms are subject to shortcomings and tradeoffs.
There are two general types of auto focus (AF) systems which exist, active auto focus and passive auto focus. In active auto focus, one or more image sensors is utilized to determine distance to the focal point, or otherwise detect focus external of the image capture lens system. Active AF systems can perform rapid focusing although they will not typically focus through windows, or in other specific applications, since sound waves and infrared light are reflected by the glass and other surfaces. In passive auto focus systems the characteristics of the viewed image are used to detect and set focus.
The majority of high-end SLR cameras currently use through-the-lens optical AF sensors, which for example, may also be utilized as light meters. The focusing ability of these modern AF systems can often be of higher precision than that achieved manually through an ordinary viewfinder.
One form of passive AF utilizes phase detection, such as by dividing the incoming light through a beam splitter into pairs of images and comparing them on an AF sensor. Two optical prisms capture the light rays coming from the opposite sides of the lens and divert it to the AF sensor, creating a simple rangefinder with a base identical to the diameter of the lens. Focus is determined in response to checking for similar light intensity patterns and phase differences calculated to determine if the object is considered in front of the focus or in back of the proper focus position.
In another type of passive AF system, contrast measurements are made within a sensor field through the lens. The system adjusts focus to maximize intensity difference between adjacent pixels which is generally indicative of correct image focus. Thus, focusing is performed until a maximum level of contrast is obtained. This form of focusing is slower than active AF, in particular when operating under dim light, but is a common method utilized in low end imaging devices. Passive systems often make poor focusing decisions in low contrast or low light conditions.
In addition, many focusing systems perform poorly when the subject is in motion, such as in regard to “action shots”, for example children at play, sports, animals in the wild, and so forth.
Accordingly, a need exists for improved depth estimation and/or auto focusing techniques which provide rapid and accurate subject-distance estimations and/or focus control under a wide range of conditions and subject motion. The present invention fulfills those needs, as well as others, and overcomes shortcomings of previous camera focus techniques.